Add databricks-agents framework for building discoverable agents#640
Open
stuagano wants to merge 8 commits intodatabrickslabs:mainfrom
Open
Add databricks-agents framework for building discoverable agents#640stuagano wants to merge 8 commits intodatabrickslabs:mainfrom
stuagano wants to merge 8 commits intodatabrickslabs:mainfrom
Conversation
|
All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits |
4320d5c to
ce0b182
Compare
Add databricks-agents, a lightweight Python framework for building discoverable AI agents on Databricks Apps. Key features: - AgentApp: FastAPI wrapper with auto-generated A2A protocol endpoints - AgentDiscovery: Workspace scanning and agent discovery - A2AClient: Agent-to-agent communication - UCAgentRegistry: Unity Catalog integration - MCPServer: Model Context Protocol support - UCFunctionAdapter: Expose UC Functions as tools The framework enables building agents with just 5 lines of code: app = AgentApp(name="my_agent", description="...", capabilities=[...]) @app.tool(description="...") async def my_tool(param: str) -> dict: ... This auto-generates: - /.well-known/agent.json (A2A protocol agent card) - /.well-known/openid-configuration (OIDC delegation) - /health (health check) - /api/mcp (MCP server) - /api/tools/* (tool endpoints) Includes: - Complete test suite - CI/CD workflows (test, publish, docs) - MkDocs documentation - 4 working examples - Apache 2.0 license Project structure: - src/databricks_agents/ - Core framework (~2000 LOC) - examples/ - Complete working examples - tests/ - Test suite - docs/ - MkDocs documentation - .github/workflows/ - CI/CD pipelines Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
ce0b182 to
c0dc300
Compare
Add the full-stack reference implementation that uses the databricks-agents framework: FastAPI backend with agent discovery, chat, search, lineage, and A2A protocol support, plus React webapp for the discovery UI.
Phase 1 — Bug fixes:
- Fix generic type extraction (List[str], Optional[int], Dict) in @app.tool
- Fix Pydantic Callable crash with ConfigDict(arbitrary_types_allowed=True)
- Replace deprecated on_event("startup") with lifespan context manager
- Replace print() with structured logging
- Fix missing setup_mcp_server export from mcp module
- Delete empty orchestration module (contradicts Direction A)
- Expand public API: UCAgentRegistry, UCAgentSpec, MCPServerConfig, etc.
Phase 2 — Test coverage (71 tests, all mocked):
- test_agent_app: tools, generics, MCP toggle, UC toggle, endpoints
- test_a2a_client: fetch, send, stream, errors, timeouts
- test_agent_discovery: workspace scanning, capabilities parsing
- test_uc_registry: register, get, list, delete, tag filtering
- test_mcp_server: JSON-RPC tools/list, tools/call, server/info
- test_uc_functions: type mapping, function conversion, discovery
Phase 3 — Documentation (17 pages):
- 4 essential pages: installation, first-agent, agent-app guide, tools guide
- 11 stub pages for remaining mkdocs nav entries
- README: UC/MCP marked done, orchestration removed from roadmap
Phase 4 — Polish:
- Add hello_agent.py minimal example
- Update full_featured_agent.py to lifespan pattern
- Add py.typed PEP 561 marker
- Wire __version__ to importlib.metadata
- Add respx dev dep and pytest asyncio_mode config
…and runtime lineage Adds the complete dashboard SPA (React + FastAPI) with three new features: - Batch UC registration: one-click "Register All in UC" from GovernanceTab - Runtime lineage: observed edges from chat traces shown as cyan dashed lines - Auto-refresh: lineage and governance tabs refresh automatically after scan Backend: register_all_agents(), ingest_trace(), POST /api/uc/register-all, POST /api/lineage/observe, enhanced scan response. Frontend: new types, API functions, fire-and-forget observe in useChat, lastScanTimestamp in useAgents for auto-refresh, updated LineageGraph with observed edge rendering, legend entries, and result banners.
Three bugs discovered during live E2E testing with real UC catalog: 1. registered_models.create() requires short name, not fully-qualified 3-part name — changed name=full_name to name=spec.name with separate catalog_name/schema_name params. 2. RegisteredModelsAPI has no set_tag/list_tags methods — replaced tag-based metadata with comment-based storage using ---AGENT_META--- JSON marker in the model comment field. 3. list_agents() failed with "Cannot have empty schema if catalog is set" — fixed to iterate schemas first, then list models per schema.
Supervisor agents now report routing decisions via _routing metadata in their MCP tool responses. The dashboard extracts this to create observed_calls_agent edges in the lineage graph (dashed cyan lines). Changes: - supervisor agent.py: track _last_routing with sub_agent name, fix mlflow compatibility (OutputItem, async tool invocation) - supervisor app.py: include _routing in route_query response - scanner.py: extract _routing from MCP tool responses into trace - governance.py: parse routing.sub_agent in ingest_trace() to create observed agent-to-agent edges
Supervisor agent now reports tables_accessed per sub-agent route, creating observed_reads_table edges in the lineage graph. Chat endpoint auto-ingests traces so edges appear without frontend round-trip. Merge function creates missing table/agent nodes.
Complete rewrite to agent platform architecture: - SDK: add_agent_card() + add_mcp_endpoints() helpers (replaces AgentApp-only pattern) - CLI: databricks-agents deploy/status/destroy/dashboard/platform commands - Deploy: multi-agent deploy from agents.yaml (topo sort, wiring, permissions) - Dashboard: discovery, governance, lineage, system builder wizard - System Builder: 4-step wizard (select, wire, configure, deploy) with dagre auto-layout, custom ReactFlow nodes/edges, async deploy with polling - Registry: Unity Catalog registration with comment-based metadata - MCP: decoupled JSON-RPC server with tool exposure - Examples: hello-world, research-agent, data-tools, supervisor patterns - Tests: 109 passing (SDK, dashboard, system builder, deploy)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add databricks-agents framework for building discoverable agents
Summary
Adds
databricks-agents, a lightweight Python framework for building discoverable AI agents on Databricks Apps. This framework makes it trivial to turn any Databricks App into a standards-compliant agent with auto-generated A2A protocol endpoints.What Problem Does This Solve?
Before: Building agents on Databricks required:
After: With databricks-agents:
Key Innovation: Agent = App
This framework treats Databricks Apps as first-class agents, enabling:
Example Usage
This automatically provides:
/.well-known/agent.json- A2A protocol agent card/.well-known/openid-configuration- OIDC delegation/health- Health check endpoint/api/mcp- MCP server for tools/api/tools/<tool_name>- Tool endpointsFramework Components
Core (
src/databricks_agents/core/)Discovery (
src/databricks_agents/discovery/)Registry (
src/databricks_agents/registry/)MCP (
src/databricks_agents/mcp/)CI/CD & Documentation
Workflows (
.github/workflows/)Documentation (
docs/)Examples
customer_research_agent.py- Basic agent with custom toolsdiscover_agents.py- Workspace agent discoverycommunicate_with_agent.py- A2A protocol communicationfull_featured_agent.py- Complete example with all featuresTesting
All tests pass with >80% coverage.
Project Structure
Why Sandbox?
This is the perfect sandbox project because it:
✅ Early-stage but valuable - Works today, provides immediate value
✅ Innovative approach - New pattern for agent building on Databricks
✅ Community-driven - Ideal for gathering feedback and contributions
✅ Low friction - 5 lines to create an agent
✅ Building block - Foundation for multi-agent systems
Graduation Path
Sandbox (0.1.x - 0.5.x)
Full Repo (1.0+) - When proven adoption:
Platform Integration - Long term:
Dependencies
All standard, well-maintained packages:
Checklist
Related Work
Builds on Databricks platform primitives:
Implements open standards:
Questions?
Happy to address any feedback or concerns! This framework solves a real gap in the Databricks agent ecosystem and provides immediate value to users building multi-agent systems.